Edit on GitHub

Tutorial

This tutorial shows the functionalities of Extremes.jl. They are illustrated by reproducing some of the results shown by Coles (2001) in An Introduction to Statistical Modeling of Extreme Values.

Before executing this tutorial, make sure to have installed the following packages:

  • Extremes.jl (of course)
  • DataFrames.jl (for using the DataFrame type)
  • Distributions.jl (for using probability distribution objects)
  • Gadfly.jl (for plotting)

and import them using the following command:

julia> using Extremes, Dates, DataFrames, Distributions, Gadfly

Model for stationary threshold exceedances

The stationary ThresholdExceedance model is illustrated using the daily rainfall accumulations at a location in south-west England from 1914 to 1962. This dataset was studied by Coles (2001) in Chapter 4.

Load the data

Loading the daily rainfall at a location in South-England:

data = load("rain")
first(data,5)

5 rows × 2 columns

DateRainfall
Date…Float64
11914-01-010.0
21914-01-022.3
31914-01-031.3
41914-01-046.9
51914-01-054.6

Plotting the data using the Gadfly package:

set_default_plot_size(14cm ,8cm)
plot(data, x=:Date, y=:Rainfall, Geom.point, Theme(discrete_highlight_color=c->nothing))
Date Jan 1, 1840 1850 1860 1870 1880 1890 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010 2020 2030 2040 Jan 1, 1800 1850 1900 1950 2000 2050 Jan 1, 1800 1850 1900 1950 2000 2050 Jan 1, 1800 1850 1900 1950 2000 2050 h,j,k,l,arrows,drag to pan i,o,+,-,scroll,shift-drag to zoom r,dbl-click to reset c for coordinates ? for help ? -125 -100 -75 -50 -25 0 25 50 75 100 125 150 175 200 225 -100 -95 -90 -85 -80 -75 -70 -65 -60 -55 -50 -45 -40 -35 -30 -25 -20 -15 -10 -5 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115 120 125 130 135 140 145 150 155 160 165 170 175 180 185 190 195 200 -100 0 100 200 -100 -90 -80 -70 -60 -50 -40 -30 -20 -10 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 Rainfall

Threshold selection

TODO

GPD parameters estimation

Let's first identify the threshold exceedances:

threshold = 30.0
df = filter(row -> row.Rainfall > threshold, data)
first(df, 5)

5 rows × 2 columns

DateRainfall
Date…Float64
11914-02-0731.8
21914-03-0832.5
31914-12-1731.8
41914-12-3044.5
51915-02-1330.5

Get the exceedances above the threshold:

df[!,:Rainfall] =  df[!,:Rainfall] .- threshold
rename!(df, :Rainfall => :Exceedance)
first(df, 5)

5 rows × 2 columns

DateExceedance
Date…Float64
11914-02-071.8
21914-03-082.5
31914-12-171.8
41914-12-3014.5
51915-02-130.5

GP parameters estimation with probability weighted moments

The GP parameter estimation with probability weighted moments is performed as follows:

julia> fm = gpfitpwm(df, :Exceedance)
pwmEVA
model :
	ThresholdExceedance
	data :		Array{Float64,1}[152]
	logscale :	ϕ ~ 1
	shape :		ξ ~ 1

θ̂  :	[1.9877399514951732, 0.19651587232938317]

The approximate covariance matrix of the parameter estimates can be obtained with the function parametervar:

julia> parametervar(fm)
2×2 Array{Float64,2}:
  0.0157177   -0.00656502
 -0.00656502   0.00683069

GP parameters estimation with maximum likelihood

The GP parameter estimation with maximum likelihood is performed as follows:

julia> fm = gpfit(df, :Exceedance)
MaximumLikelihoodEVA
model :
	ThresholdExceedance
	data :		Array{Float64,1}[152]
	logscale :	ϕ ~ 1
	shape :		ξ ~ 1

θ̂  :	[2.006896498380506, 0.1844926991237574]

The approximate covariance matrix of the parameter estimates can be obtained with the function parametervar:

julia> parametervar(fm)
2×2 Array{Float64,2}:
  0.0165972   -0.00880429
 -0.00880429   0.0102416

GP parameters estimation with the Bayesian method

The GP parameter estimation with the Bayesian method is performed as follows:

julia> fm = gpfitbayes(df, :Exceedance)

Progress:   0%|                                         |  ETA: 0:18:49
Progress:  22%|█████████▎                               |  ETA: 0:00:02
Progress:  37%|███████████████                          |  ETA: 0:00:01
Progress:  40%|████████████████▍                        |  ETA: 0:00:01
Progress:  51%|████████████████████▉                    |  ETA: 0:00:01
Progress:  64%|██████████████████████████▍              |  ETA: 0:00:01
Progress:  75%|██████████████████████████████▉          |  ETA: 0:00:00
Progress:  89%|████████████████████████████████████▍    |  ETA: 0:00:00
Progress: 100%|█████████████████████████████████████████| Time: 0:00:01
BayesianEVA
model :
	ThresholdExceedance
	data :		Array{Float64,1}[152]
	logscale :	ϕ ~ 1
	shape :		ξ ~ 1

sim :
	Mamba.Chains
	Iterations :		2001:5000
	Thinning interval :	1
	Chains :		1
	Samples per chain :	3000
	Value :			Array{Float64,3}[3000,2,1]
Prior

Currently, only the improper uniform prior is implemented, i.e. \[ f_{(ϕ,ξ)}(ϕ,ξ) ∝ 1. \] It yields to a proper posterior as long as the sample size is larger than 2 (Northrop and Attalides, 2016).

Sampling scheme

Currently, the No-U-Turn Sampler extension (Hoffman and Gelman, 2014) to Hamiltonian Monte Carlo (Neel, 2011, Chapter 5) is implemented for simulating an autocorrelated sample from the posterior distribution.

The approximate covariance matrix of the parameter estimates can be obtained with the function parametervar:

julia> parametervar(fm)
2×2 Array{Float64,2}:
  0.0161428   -0.00873375
 -0.00873375   0.0106877

Return level estimation

With the ThresholdExceedance structure, the returnlevel function requires several arguments to calculate the T-year return level:

  • the threshold value;
  • the number of total observation (below and above the threshold);
  • the number of observations per year;
  • the return period T;
  • the confidence level for computing the confidence interval.

The function uses the Peaks-Over-Threshold model definition (Coles, 2001, Chapter 4) for computing the T-year return level.

For the rainfall example, the 100-year return level can be estimated as follows:

fm = gpfit(df, :Exceedance)
r = returnlevel(fm, threshold, size(data,1), 365, 100, .95)
ReturnLevel
fittedmodel :
		MaximumLikelihoodEVA
		model :
			ThresholdExceedance
			data :		Array{Float64,1}[152]
			logscale :	ϕ ~ 1
			shape :		ξ ~ 1

		θ̂  :	[2.006896498380506, 0.1844926991237574]
returnperiod :	100
value :		Array{Float64,1}[1]
cint :		Array{Array{Float64,1},1}[1]

where the value can be accessed with

julia> r.value
1-element Array{Float64,1}:
 106.32558691303024

and where the corresponding confidence interval can be accessed with

julia> r.cint
1-element Array{Array{Float64,1},1}:
 [65.48163774428642, 147.16953608177405]
Note

In this example of a stationary model, the function returns a unit dimension vector for the return level and a vector containing only one vector for the confidence interval. The reason is that the function always returns the same type in the stationary and non-stationary case. The function is therefore type-stable allowing better performance of code execution.

To get the scalar return level in the stationary case, the following command can be used:

julia> r.value[]
106.32558691303024

To get the scalar confidence interval in the stationary case, the following command can be used:

julia> r.cint[]
2-element Array{Float64,1}:
  65.48163774428642
 147.16953608177405

Probability weighted moments estimation

Probability weighted moments estimation of the GEV parameters can also be performed by using the gevfitpwm function. All the methods also apply to the pwmEVA object.

julia> fm = gpfitpwm(df, :Exceedance)
pwmEVA
model :
	ThresholdExceedance
	data :		Array{Float64,1}[152]
	logscale :	ϕ ~ 1
	shape :		ξ ~ 1

θ̂  :	[1.9877399514951732, 0.19651587232938317]

Bayesian estimation

Bayesian estimation of the GEV parameters can also be performed by using the gevfitbayes function. All the methods also apply to the `BayesianEVA object.

julia> fm = gpfitbayes(df, :Exceedance)

Progress:  12%|█████▏                                   |  ETA: 0:00:01
Progress:  22%|████████▉                                |  ETA: 0:00:01
Progress:  36%|██████████████▋                          |  ETA: 0:00:01
Progress:  49%|████████████████████▏                    |  ETA: 0:00:00
Progress:  59%|████████████████████████▏                |  ETA: 0:00:00
Progress:  73%|█████████████████████████████▉           |  ETA: 0:00:00
Progress:  85%|██████████████████████████████████▉      |  ETA: 0:00:00
Progress:  96%|███████████████████████████████████████▍ |  ETA: 0:00:00
Progress: 100%|█████████████████████████████████████████| Time: 0:00:00
BayesianEVA
model :
	ThresholdExceedance
	data :		Array{Float64,1}[152]
	logscale :	ϕ ~ 1
	shape :		ξ ~ 1

sim :
	Mamba.Chains
	Iterations :		2001:5000
	Thinning interval :	1
	Chains :		1
	Samples per chain :	3000
	Value :			Array{Float64,3}[3000,2,1]

Model for dependent data

The stationary ThresholdExceedance model is illustrated using the daily rainfall accumulations at a location in south-west England from 1914 to 1962. This dataset was studied by Coles (2001) in Chapter 4.

Load the data

Loading the daily rainfall at a location in South-England:

data = load("wooster")
first(data,5)

5 rows × 2 columns

DateTemperature
Date…Int64
11983-01-0123
21983-01-0229
31983-01-0319
41983-01-0414
51983-01-0527

Plotting the data using the Gadfly package:

plot(data, x=:Date, y=:Temperature, Geom.point, Theme(discrete_highlight_color=c->nothing))
Date Jan 1, 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 Jan 1, 1975 1980 1985 1990 1995 Jan 1, 1975 1980 1985 1990 1995 Jan 1, 1975 1980 1985 1990 1995 h,j,k,l,arrows,drag to pan i,o,+,-,scroll,shift-drag to zoom r,dbl-click to reset c for coordinates ? for help ? -150 -125 -100 -75 -50 -25 0 25 50 75 100 125 150 175 200 -125 -120 -115 -110 -105 -100 -95 -90 -85 -80 -75 -70 -65 -60 -55 -50 -45 -40 -35 -30 -25 -20 -15 -10 -5 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115 120 125 130 135 140 145 150 155 160 165 170 175 -200 -100 0 100 200 -130 -120 -110 -100 -90 -80 -70 -60 -50 -40 -30 -20 -10 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 Temperature 
df = copy(data)
df[!,:Temperature] = -data[:,:Temperature]
filter!(row -> month(row.Date) ∈ (1,2,11,12), df)
plot(df, x=:Date, y=:Temperature, Geom.point)
Date Jan 1, 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 Jan 1, 1975 1980 1985 1990 1995 Jan 1, 1975 1980 1985 1990 1995 Jan 1, 1975 1980 1985 1990 1995 h,j,k,l,arrows,drag to pan i,o,+,-,scroll,shift-drag to zoom r,dbl-click to reset c for coordinates ? for help ? -160 -140 -120 -100 -80 -60 -40 -20 0 20 40 60 80 100 120 -140 -135 -130 -125 -120 -115 -110 -105 -100 -95 -90 -85 -80 -75 -70 -65 -60 -55 -50 -45 -40 -35 -30 -25 -20 -15 -10 -5 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 -200 -100 0 100 -140 -135 -130 -125 -120 -115 -110 -105 -100 -95 -90 -85 -80 -75 -70 -65 -60 -55 -50 -45 -40 -35 -30 -25 -20 -15 -10 -5 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 Temperature

Declustering the threshold exceedances

threshold = -10
cluster = getcluster(df[:,:Temperature], -10, runlength=4)
julia> typeof(cluster)
Array{Cluster,1}

GPD parameters estimation

Let's first identify the threshold exceedances:

threshold = 30.0
df = filter(row -> row.Rainfall > threshold, data)
first(df, 5)

5 rows × 2 columns

DateRainfall
Date…Float64
11914-02-0731.8
21914-03-0832.5
31914-12-1731.8
41914-12-3044.5
51915-02-1330.5

Get the exceedances above the threshold:

df[!,:Rainfall] =  df[!,:Rainfall] .- threshold
rename!(df, :Rainfall => :Exceedance)
first(df, 5)

5 rows × 2 columns

DateExceedance
Date…Float64
11914-02-071.8
21914-03-082.5
31914-12-171.8
41914-12-3014.5
51915-02-130.5

Generalized Pareto parameter estimation by maximum likelihood:

julia> fm = gpfit(df, :Exceedance)
MaximumLikelihoodEVA
model :
	ThresholdExceedance
	data :		Array{Float64,1}[152]
	logscale :	ϕ ~ 1
	shape :		ξ ~ 1

θ̂  :	[2.006896498380506, 0.1844926991237574]
Note

The function returns the estimates of the log-scale parameter $\phi = \log \sigma$.

Return level estimation

With the ThresholdExceedance structure, the returnlevel function requires several arguments to calculate the T-year return level:

  • the threshold value;
  • the number of total observation (below and above the threshold);
  • the number of observations per year;
  • the return period T;
  • the confidence level for computing the confidence interval.

The function uses the Peaks-Over-Threshold model definition (Coles, 2001, Chapter 4) for computing the T-year return level.

For the rainfall example, the 100-year return level can be estimated as follows:

julia> r = returnlevel(fm, threshold, size(data,1), 365, 100, .95)
ReturnLevel
fittedmodel :
		MaximumLikelihoodEVA
		model :
			ThresholdExceedance
			data :		Array{Float64,1}[152]
			logscale :	ϕ ~ 1
			shape :		ξ ~ 1

		θ̂  :	[2.006896498380506, 0.1844926991237574]
returnperiod :	100
value :		Array{Float64,1}[1]
cint :		Array{Array{Float64,1},1}[1]

where the value can be accessed with

julia> r.value
1-element Array{Float64,1}:
 106.32558691303024

and where the corresponding confidence interval can be accessed with

julia> r.cint
1-element Array{Array{Float64,1},1}:
 [65.48163774428642, 147.16953608177405]
Note

In this example of a stationary model, the function returns a unit dimension vector for the return level and a vector containing only one vector for the confidence interval. The reason is that the function always returns the same type in the stationary and non-stationary case. The function is therefore type-stable allowing better performance of code execution.

To get the scalar return level in the stationary case, the following command can be used:

julia> r.value[]
106.32558691303024

To get the scalar confidence interval in the stationary case, the following command can be used:

julia> r.cint[]
2-element Array{Float64,1}:
  65.48163774428642
 147.16953608177405

Probability weighted moments estimation

Probability weighted moments estimation of the GEV parameters can also be performed by using the gevfitpwm function. All the methods also apply to the pwmEVA object.

julia> fm = gpfitpwm(df, :Exceedance)
pwmEVA
model :
	ThresholdExceedance
	data :		Array{Float64,1}[152]
	logscale :	ϕ ~ 1
	shape :		ξ ~ 1

θ̂  :	[1.9877399514951732, 0.19651587232938317]

Bayesian estimation

Bayesian estimation of the GEV parameters can also be performed by using the gevfitbayes function. All the methods also apply to the `BayesianEVA object.

julia> fm = gpfitbayes(df, :Exceedance)

Progress:  12%|████▉                                    |  ETA: 0:00:01
Progress:  22%|█████████▎                               |  ETA: 0:00:01
Progress:  36%|██████████████▊                          |  ETA: 0:00:01
Progress:  49%|████████████████████                     |  ETA: 0:00:00
Progress:  62%|█████████████████████████▌               |  ETA: 0:00:00
Progress:  76%|███████████████████████████████          |  ETA: 0:00:00
Progress:  86%|███████████████████████████████████▍     |  ETA: 0:00:00
Progress:  99%|████████████████████████████████████████▊|  ETA: 0:00:00
Progress: 100%|█████████████████████████████████████████| Time: 0:00:00
BayesianEVA
model :
	ThresholdExceedance
	data :		Array{Float64,1}[152]
	logscale :	ϕ ~ 1
	shape :		ξ ~ 1

sim :
	Mamba.Chains
	Iterations :		2001:5000
	Thinning interval :	1
	Chains :		1
	Samples per chain :	3000
	Value :			Array{Float64,3}[3000,2,1]

Model for non-stationary data

Coles(2001, Chapter 6)